Image processing apparatus

ABSTRACT

In one embodiment, an element  106  transforms non-polarized light into plane polarized light with an arbitrary plane of polarization. A synchronizer  112  gives the element  106  an instruction to rotate the plane of polarization, thereby getting the plane of polarization of the illumination rotated and casting that polarized illumination toward the object. At the same time, the synchronizer  112  sends a shooting start signal to an image sensor  110,  thereby getting video. The synchronizer  112  performs these processing steps multiple times. A captured video signal is sent to an image processing processor  108.  Candidates for the azimuth angle of a surface normal are obtained based on an intensity maximizing angle image, the zenith angle of the surface normal is obtained based on a degree of intensity modulation image, and the ambiguity of the azimuth angle is solved, thereby generating a surface groove normal image.

This is a continuation of International Application No.PCT/JP2011/003931, with an international filing date of Jul. 8, 2011,which claims priority of Japanese Patent Application No. 2010-164074,filed on Jul. 21, 2010, the contents of which are hereby incorporated byreference.

BACKGROUND

1. Technical Field

The present application relates to an image processing apparatus thatcan obtain surface microfacet information that surpasses information tobe normally obtained by an image sensor from a two-dimensional lightintensity image.

2. Related Art

In the field of endoscopes that capture an image of an organism's organby irradiating the surface of the organ, which is covered with asemi-transparent mucosa, with light, the surface texture and an image ofa blood vessel under the surface need to be checked with regularreflection (i.e., specular reflection) from the surface avoided. To dothat, a polarizing endoscope that uses polarized light and polarizedimage capturing has been proposed. For example, Patent Document No. 1discloses an endoscope that includes a polarized light source sectionthat irradiates an object with light having a particular polarizationcomponent and a light receiving section and that generates a shapevariation image representing a variation in the surface shape of theobject. The light receiving section of that endoscope receives lightwith a particular polarization component that is included in the lightreturning from the object and light with a different polarizationcomponent from the particular one that is also included in the returninglight. The image capturing section disclosed in Patent Document No. 1includes an RGB color mosaic and polarizers, which are arranged so thattheir polarization transmission axes face three different directions.Patent Document No. 1 says that to allow the viewer to easily recognizethe surface microfacets of the mucosa, in particular, a polarizationproperty calculating section calculates a polarization orientation andcan generate a two-dimensional distribution of surface tilt information.

CITATION LIST Patent Literature

-   -   Patent Document No. 1: Japanese Laid-Open Patent Publication No.        2009-246770    -   Patent Document No. 2: Japanese Laid-Open Patent Publication No.        11-313242    -   Patent Document No. 3: United States Laid-Open Patent        Publication No. 2009/0079982    -   Patent Document No. 4: Japanese Laid-Open Patent Publication No.        2007-86720    -   Patent Document No. 5: PCT International Application Publication        No. 2008/149489

Non-Patent Literature

-   -   Non-Patent Document No. 1: Nicolas Lefaudeux et al., “Compact        and robust linear Stokes polarization camera”, Proc. of SPIE,        Vol. 6972, 69720B, Polarization: Measurement, Analysis, and        Remote Sensing VIII (2008);    -   Non-Patent Document No. 2: Gary A. Atkinson, Edwin R. Hancock:        “Recovery of Surface Orientation from Diffuse Polarization”,        IEEE Transactions of Image Processing, Vol. 15, No. 6, June        2006, pp. 1653-1664.    -   Non-Patent Document No. 3: Daisuke Miyazaki and Katsushi        Ikeuchi, “Measuring Surface Shape of Transparent Objects from        the Analysis of Parabolic Curves and Polarization”, Journal of        Information Processing, Vol.44, No. SIG 9 (CVIM 7), July 2003,        pp. 86-93

SUMMARY

The prior art technique needs further improvement in view of imagequality.

One non-limiting, and exemplary embodiment provides a technique to animage processing apparatus that can obtain polarization information on apixel-by-pixel basis and that can get information about the object'ssurface microfacets based on that polarization information.

In one general aspect, an image processing apparatus disclosed hereincomprises: a polarized light source section that sequentiallyilluminates an object with three or more kinds of plane polarized lightrays, of which the planes of polarization have mutually differentangles; an image capturing section that sequentially captures an imageof the object that is being illuminated with each of the three or morekinds of plane polarized light rays and directly receives light that isreturning from the object by way of no polarizers, thereby getting anintensity value; a varying intensity processing section that obtains arelation between the angle of the plane of polarization and theintensity value of each pixel based on a signal representing theintensity value supplied from the image capturing section, therebygenerating an intensity maximizing angle image that is defined by theangle of the plane of polarization that maximizes the intensity valuewith respect to each said pixel and a degree of intensity modulationimage that is defined by the ratio of the amplitude of variation in theintensity value caused by the change of the plane of polarization to anaverage intensity value with respect to each said pixel; and a normalestimating section that estimates, based on the intensity maximizingangle image and the degree of intensity modulation image, a normal to atilted surface in a V-groove on the object's surface on a pixel-by-pixelbasis.

In another aspect, an image processing method disclosed herein comprisesthe steps of: sequentially illuminating an object with three or morekinds of plane polarized light rays, of which the planes of polarizationhave mutually different angles; sequentially capturing an image of theobject when the object is being illuminated with each of the three ormore kinds of plane polarized light rays and directly receiving lightthat is returning from the object by way of no polarizers, therebygetting an intensity value; obtaining a relation between the angle ofthe plane of polarization and the intensity value of each pixel, therebygenerating an intensity maximizing angle image that is defined by theangle of the plane of polarization that maximizes the intensity valuewith respect to each said pixel and a degree of intensity modulationimage that is defined by the ratio of the amplitude of variation in theintensity value caused by the change of the plane of polarization to anaverage intensity value with respect to each said pixel; and estimating,based on the intensity maximizing angle image and the degree ofintensity modulation image, a normal to a tilted surface in a V-grooveon the object's surface on a pixel-by-pixel basis.

In another aspect, an image processing processor disclosed hereinreceives a plurality of polarized images, where the plane ofpolarization of a plane polarized light ray with which an object isilluminated has three or more different angles, and estimates, throughimage processing, a normal to a tilted surface in a V-groove on theobject's surface on a pixel-by-pixel basis. The image processingprocessor performs the steps of: obtaining a relation between the angleof the plane of polarization and the intensity value of each pixel basedon the polarized images, thereby generating an intensity maximizingangle image that is defined by the angle of the plane of polarizationthat maximizes the intensity value with respect to each said pixel and adegree of intensity modulation image that is defined by the ratio of theamplitude of variation in the intensity value caused by the change ofthe plane of polarization to an average intensity value with respect toeach said pixel; and estimating, based on the intensity maximizing angleimage and the degree of intensity modulation image, a normal to thetilted surface in the V-groove on the object's surface on apixel-by-pixel basis.

These general and specific aspects may be implemented using a system, amethod, and a computer program, and any combination of systems, methods,and computer programs.

Additional benefits and advantages of the disclosed embodiments will beapparent from the specification and Figures. The benefits and/oradvantages may be individually provided by the various embodiments andfeatures of the specification and drawings disclosure, and need not allbe provided in order to obtain one or more of the same.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A Illustrates an exemplary configuration for an image processingapparatus according to the present disclosure.

FIG. 1B shows polarization states of a polarized light source.

FIG. 1C illustrates a configuration for an image processing apparatus asa first embodiment of the present disclosure.

FIG. 2 shows how a plane of polarization control element operates.

FIG. 3 shows how to define the angle of a plane of polarization.

FIGS. 4( a) and 4(b) illustrate an exemplary arrangement ofphotosensitive cells in an image sensor for use in the first embodimentof the present disclosure.

FIG. 5 illustrates a configuration for another image processingapparatus that is designed to obtain a color image and a polarizedimage.

FIG. 6 Illustrates an arrangement of photosensitive cells in thepolarized image sensor of the image processing apparatus shown in FIG.5.

FIG. 7A illustrates how intensity pattern images change as thepolarization plane of a polarized light source rotates.

FIG. 7B schematic representations illustrating how intensity patternimages change as the polarization plane of a polarized light sourcerotates.

FIGS. 8( a) and 8(b) illustrate how incoming light that has comedirectly from over an object is incident on the object's surface andreflected once.

FIG. 9 is a graph showing how the Fresnel reflectances of P- and S-waveenergies change with the angle of incidence (that is represented as theabscissa).

FIG. 10A is a graph showing how the intensity value of a pixel varies asthe plane of polarization of polarized light is rotated.

FIG. 10B is a photograph showing the surface shape of a sample that wasused to get the data shown in the graph of FIG. 10A.

FIG. 10C schematically illustrates the surface shape shown in FIG. 10B.

FIG. 11( a) shows the polarization directions of polarized light sourcesand FIG. 11( b) shows how the light intensity varies according to thepolarized light source.

FIGS. 12( a) and 12(b) illustrate how the intensity of polarizedreflected light varies due to interreflection.

FIGS. 13( a), 13(b) and 13(c) illustrate a groove on the object'ssurface as viewed from right over that surface.

FIG. 14A illustrates a situation where polarized light is incident on agroove at ψ=0 degrees.

FIG. 14B illustrates a situation where reflected light is producedparallel and perpendicularly to the azimuth angle ψ of the groove in thestate shown in FIG. 14A.

FIG. 15 illustrates a situation where non-polarized light is incident ona groove and reflected light is produced parallel and perpendicularly tothe azimuth angle ψ of the groove.

FIG. 16 is a diagram illustrating a configuration for an imageprocessing processor according to the first embodiment of the presentdisclosure.

FIG. 17 shows how to fit a cosine function based on samples of thepolarized light intensities of four different kinds of polarized lightsources.

FIG. 18A shows a relation between the XYZ components of a surfacenormal, the azimuth angle ψ and the zenith angle θ.

FIG. 18B shows a relation between a surface normal N, a light sourcevector L, a viewpoint vector V and a bisector vector H.

FIG. 19A illustrates a situation where an intensity gradient vector isused to solve a 180 degree ambiguity of the azimuth angle of a normal.

FIG. 19B illustrates a situation where a degree of intensity modulationgradient vector is used to solve the 180 degree ambiguity of the azimuthangle of a normal.

FIG. 19C is a flowchart showing how to solve the ambiguity of theazimuth angle ψ of a normal.

FIG. 20A is a graph plotting theoretical values that show relationsbetween the angle of incidence θ and the degree of polarization producedby Fresnel reflection.

FIG. 20B indicates the search range of a zenith angle θ.

FIG. 20C is a flowchart showing how to determine the zenith angle θ.

FIG. 21( a) shows the result of an experiment in which a lenticular lensplate was used as an object and 21(b) schematically illustrates a partof the portion 21(a) on a larger scale.

FIG. 22 illustrates how once reflection and twice reflection areproduced at a cross section of a lenticular lens plate.

FIG. 23 shows the result of an experiment on the light intensity anddegree of intensity modulation YD of a lenticular lens plate.

FIG. 24A illustrates, as an example, a stellar object with grooves.

FIG. 24B schematically illustrates the object shown in FIG. 24A.

FIG. 25 shows results of estimation of the azimuth angles ψ of normalvectors with respect to those grooves of the stellar object.

FIG. 26 shows results of estimation of the zenith angles θ of normalvectors with respect to those grooves of the stellar object.

FIG. 27A shows results of experiments in which images were generated byilluminating the normal images of the stellar object in four differentdirections.

FIG. 27B schematically illustrates the images shown in FIG. 27A.

FIG. 28 illustrates a configuration for an image processing apparatusaccording to a second embodiment of the present disclosure.

FIG. 29 shows the optimized characteristic of a spectral filter.

FIG. 30A illustrates a configuration according to a third embodiment ofthe present disclosure.

FIG. 30B illustrates the appearance of the third embodiment of thepresent disclosure.

DETAILED DESCRIPTION

The present inventors discovered via experiments that the imagecapturing section disclosed in Cited Reference #1 could not get accuratepolarization information on a pixel-by-pixel basis. On top of that, anoticeable moire pattern would be generated in the polarized image dueto interference with the spatial frequency of the object and part of thecolor mosaic would turn into a polarization mosaic, thus debasing thequality of a full-color image reproduced, too.

In one general aspect, an image processing apparatus disclosed hereincomprises: a polarized light source section that sequentiallyilluminates an object with three or more kinds of plane polarized lightrays, of which the planes of polarization have mutually differentangles; an image capturing section that sequentially captures an imageof the object that is being illuminated with each of the three or morekinds of plane polarized light rays and directly receives light that isreturning from the object by way of no polarizers, thereby getting anintensity value; a varying intensity processing section that obtains arelation between the angle of the plane of polarization and theintensity value of each pixel based on a signal representing theintensity value supplied from the image capturing section, therebygenerating an intensity maximizing angle image that is defined by theangle of the plane of polarization that maximizes the intensity valuewith respect to each said pixel and a degree of intensity modulationimage that is defined by the ratio of the amplitude of variation in theintensity value caused by the change of the plane of polarization to anaverage intensity value with respect to each said pixel; and a normalestimating section that estimates, based on the intensity maximizingangle image and the degree of intensity modulation image, a normal to atilted surface in a V-groove on the object's surface on a pixel-by-pixelbasis.

In one embodiment, the normal estimating section includes: an azimuthangle processing section that obtains candidates for the azimuth angleof the normal based on the intensity maximizing angle image; a zenithangle processing section that obtains the zenith angle of the normalbased on the degree of intensity modulation image; and an azimuth angleambiguity processing section that chooses one among those candidates forthe azimuth angle of the normal.

In one embodiment, the image processing apparatus includes a normalimage generating section that generates an image of the normal that hasbeen estimated by the normal estimating section.

In one embodiment, the azimuth angle ambiguity processing sectionchooses one among those candidates for the azimuth angle of the normalby reference to either a non-polarized light intensity imagecorresponding to an image under non-polarized light or the degree ofintensity modulation image.

In one embodiment, the varying intensity processing section addstogether the multiple light intensity images that have been obtained bythe image capturing section and calculates their average, therebygenerating and giving the non-polarized light intensity image to theazimuth angle ambiguity processing section.

In one embodiment, the azimuth angle ambiguity processing sectionchooses one among those candidates for the azimuth angle of the normalbased on at least one of the respective spatial gradient vectors of thenon-polarized light intensity image and the degree of intensitymodulation image.

In one embodiment, the polarized light source section and the imagecapturing section are attached to an endoscope.

In one embodiment, the polarized light source section gets non-polarizedlight transmitted through a plane of polarization changer that is ableto change planes of polarization, thereby radiating plane polarizedlight rays, of which the plane of polarization sequentially changes intoone of three or more different types after another.

In one embodiment, an angle of 15 degrees or less is defined between therespective optical axes of the polarized light source and the imagecapturing section.

In one embodiment, the image capturing section includes either amonochrome image sensor or a color image sensor.

In one embodiment, the image processing apparatus includes: anilluminating direction setting section that virtually changes freely theilluminating direction of the object; and a light intensity imagegenerating section that generates, based on the output of the normalestimating section, a light intensity image of the object beingilluminated in the illuminating direction.

In one embodiment, the polarized light source section includes, on itsoutput stage, a spectral filter that transmits light falling within awavelength range associated with a reflectance at which the spectralreflectance characteristic at the object's surface reaches a localminimum.

In one embodiment, the polarized light source section includes: a ringlight source that radiates non-polarized light; and a ring plane ofpolarization changer that transforms the non-polarized light radiatedfrom the ring light source into the plane polarized light ray and thatis able to change the angle of the plane of polarization of the planepolarized light ray sequentially.

In another aspect, an image processing method disclosed herein comprisesthe steps of: sequentially illuminating an object with three or morekinds of plane polarized light rays, of which the planes of polarizationhave mutually different angles; sequentially capturing an image of theobject when the object is being illuminated with each of the three ormore kinds of plane polarized light rays and directly receiving lightthat is returning from the object by way of no polarizers, therebygetting an intensity value; obtaining a relation between the angle ofthe plane of polarization and the intensity value of each pixel, therebygenerating an intensity maximizing angle image that is defined by theangle of the plane of polarization that maximizes the intensity valuewith respect to each said pixel and a degree of intensity modulationimage that is defined by the ratio of the amplitude of variation in theintensity value caused by the change of the plane of polarization to anaverage intensity value with respect to each said pixel; and estimating,based on the intensity maximizing angle image and the degree ofintensity modulation image, a normal to a tilted surface in a V-grooveon the object's surface on a pixel-by-pixel basis.

In another aspect, an image processing processor disclosed hereinreceives a plurality of polarized images, where the plane ofpolarization of a plane polarized light ray with which an object isilluminated has three or more different angles, and estimates, throughimage processing, a normal to a tilted surface in a V-groove on theobject's surface on a pixel-by-pixel basis. The image processingprocessor performs the steps of: obtaining a relation between the angleof the plane of polarization and the intensity value of each pixel basedon the polarized images, thereby generating an intensity maximizingangle image that is defined by the angle of the plane of polarizationthat maximizes the intensity value with respect to each said pixel and adegree of intensity modulation image that is defined by the ratio of theamplitude of variation in the intensity value caused by the change ofthe plane of polarization to an average intensity value with respect toeach said pixel; and estimating, based on the intensity maximizing angleimage and the degree of intensity modulation image, a normal to thetilted surface in the V-groove on the object's surface on apixel-by-pixel basis.

As shown in FIG. 1A, an exemplary image processing apparatus accordingto the present invention includes a polarized light source section 120,an image capturing section 140, a varying intensity processing section160, and a normal estimating section 170. The varying intensityprocessing section 160 and the normal estimating section 170 areincluded in an image processing section 150.

The polarized light source section 120 sequentially illuminates anobject 100 with three or more kinds of plane polarized light rays, ofwhich the planes of polarization have mutually different angles. On thesurface of the object 100 of shooting according to the presentinvention, there are multiple grooves 100 a. If the object 100 is thesurface of an organism's organ, for example, multiple grooves areobserved. A plane polarized light ray is reflected by the groove 100 aon the surface of the object 100 and then incident on the imagecapturing section 140. When the object 100 is being illuminated witheach of the three or more kinds of plane polarized light rays, the imagecapturing section 140 shoots the object 100 sequentially. In themeantime, the image capturing section 140 receives the light returningfrom the object by way of no polarizers, thereby getting an intensityvalue.

In this description, the “returning light” refers herein to a part ofthe light that has been emitted from the polarized light source section120, reflected from the surface of the object 100 and then incident onthe image capturing section 140. To illuminate the inside of the grooves100 a on the surface of the object 100 with the light that has beenradiated from the polarized light source 120, the angle defined betweenthe respective optical axes of the polarized light source section 120and the image capturing section 140 may be not too large. Specifically,that angle defined between the respective optical axes of the polarizedlight source section 120 and the image capturing section 140 may be setto be 15 degrees or less.

FIG. 1B is a perspective view schematically showing the polarizationdirections of three kinds of plane polarized light rays, of which theplanes of polarization have mutually different angles. The threepolarization states 10, 12 and 14 illustrated in FIG. 1B have planes ofpolarization that have mutually different angles. Inside each of thesecircles schematically illustrating the respective polarization states10, 12 and 14 in FIG. 1B, shown is a double-headed arrow, whichindicates the vibration direction of the electric vector that definesthe plane of polarization of a plane polarized light ray.

The XYZ coordinates shown in FIG. 1B are of the right-handed system. Inthis description, the X- and Y-axes are defined in the plane of theimage captured by the image capturing section 140, and the direction ofthe Z-axis is defined to be the viewing direction (i.e., the opticalaxis direction). The plane of polarization of a plane polarized lightray is a plane that is parallel to the vibrating electric field vectorand that includes the optical axis. If this coordinate system isadopted, the electric field vector vibration direction of the planepolarized light ray is parallel to the XY plane. That is why the angle(θ I) of the plane of polarization is defined to be the angle formed bythe polarization direction (i.e., the electric vector vibrationdirection) with respect to the positive X-axis direction. This angle 74I will be described in detail later with reference to FIG. 3.

According to the present invention, the polarized light source section120 sequentially illuminates the object 100 with three or more kinds ofplane polarized light rays, of which the planes of polarization havemutually different angles. And while the object 100 is being illuminatedwith each of the three or more kinds of plane polarized light rays, theimage capturing section 140 shoots the object 100 sequentially. In themeantime, the image capturing section 140 receives the light returningfrom the object by way of no polarizers, thereby getting an intensityvalue.

Now let's go back to FIG. 1A. The varying intensity processing section160 obtains a relation between the angle of the plane of polarizationand the intensity value of each pixel based on a signal representing theintensity value supplied from the image capturing section 140, therebygenerating an “intensity maximizing angle image” and a “degree ofintensity modulation image”. In this description, the “intensitymaximizing angle image” is an image that is defined by the angle of theplane of polarization that maximizes the intensity value with respect toeach of the pixels that form the image captured. For example, if theintensity value of a pixel P (x, y) that is defined by a set ofcoordinates (x, y) becomes maximum when the object 100 is illuminatedwith a plane polarized light ray, of which the plane of polarization hasan angle of 45 degrees, then an intensity maximizing angle of 45 degreesis set with respect to that pixel P (x, y). A single “intensitymaximizing angle image” is formed by setting such an intensitymaximizing angle value for each pixel. On the other hand, the “degree ofintensity modulation image” is an image that is defined by the ratio ofthe amplitude of variation in the intensity value caused by the changeof the plane of polarization to an average intensity value with respectto each pixel. Specifically, if the degree of intensity modulation withrespect to a certain pixel P (x, y) is 0.3, then the value of 0.3 is setfor that pixel P (x, y). A single “degree of intensity modulation image”is formed by setting such a degree of intensity modulation value foreach pixel.

As can be seen, in this description, an “image” refers herein to notonly a light intensity image to be directly sensible to human vison butalso any arrangement of numerical values that are allocated torespective pixels. For example, if a single “intensity maximizing angleimage” is displayed, the image can be displayed with lightness definedby the intensity maximizing angle value that has been set for each pixelof that intensity maximizing angle image. The intensity maximizing angleimage represented in this manner does include a bright and dark patternthat is sensible to human eyes but that is different from an ordinarylight intensity image representing the object's intensity. It should benoted that the data itself that represents any of various kinds of“images” will also be sometimes referred to herein as an “image” for thesake of simplicity.

The normal estimating section 170 shown in FIG. 1A estimates, based onthe intensity maximizing angle image and the degree of intensitymodulation image, a normal to a tilted surface in a V-groove 100 a onthe object's (100) surface on a pixel-by-pixel basis. When the V-groove100 a is viewed straight on, the azimuth angle of a normal to a tiltedsurface in that V-groove 100 a is perpendicular to the direction inwhich the V-groove 100 a runs. In one embodiment of the presentinvention, first of all, the direction of the V-groove 100 a, i.e., theazimuth angle of a normal to a tilted surface in the V-groove 100 a, isdetermined. After that, the zenith angle of the normal to the tiltedsurface in the V-groove 100 a is determined. It will be described indetail later exactly on what principle the normal estimating section 170of the present invention estimates the normal to the tilted surface inthe V-groove 100 a on a pixel-by-pixel basis.

Embodiment 1

FIG. 1C schematically illustrates an overall configuration for an imageprocessing apparatus as a first embodiment of the present invention.

This image processing apparatus includes an endoscope 101 and acontroller 102. The endoscope 101 includes a tip portion 113 with animage capturing sensor and an inserting portion 103 with a light guide105 and a video signal line 111. The inserting portion 103 of theendoscope 101 actually has a structure that is more elongatedhorizontally than in FIG. 1C and that can be bent flexibly. Even whenbent, the light guide 106 can also propagate light. It should be notedthat there are two types of endoscopes, that is, a flexible scope with aflexible inserting portion 103 such as the one of this embodiment and arigid scope with an inflexible inserting portion. In the rigid scopethat is the other type of an endoscope, its inserting portion 103 has astructure for guiding returning light to an image sensor, which islocated behind it, using a relay optical system, for example. Thepresent invention is applicable to both a flexible scope and a rigidscope.

The controller 102 includes a light source 104, an image processingprocessor 108 and a synchronizer 112. The white non-polarized light thathas been emitted from the light source 104 is guided through the lightguide 105 to a plane of polarization control element 106 of the tipportion 113. The plane of polarization control element 106 may be madeup of a polarizer and a liquid crystal element and can transform thenon-polarized light into plane polarized light with an arbitrary planeof polarization using a voltage.

The plane of polarization control element 106 is a device that canrotate the plane of polarization using a liquid crystal material. Itsexemplary configurations are already disclosed in Patent Documents Nos.2 and 3, Non-Patent Document No. 1 and so on. The plane ofpolarizationcontrol element 106 may be implemented as a voltageapplication type liquid crystal device that includes a ferroelectricliquid crystal material, a polarization film and a quarter-wave plate incombination. The plane of polarization control element 106 transformsthe non-polarized light that has been produced by the light source 104and then transmitted through the light guide 105 into plane polarizedlight that has a plane of polarization at an arbitrary angle.

The synchronizer 112 gives the plane of polarization control element 106an instruction to rotate the plane of polarization, thereby getting theplane of polarization of the illumination rotated. And that polarizedillumination is cast toward the object through an illuminating lens 107.At the same time, the synchronizer 112 sends a shooting start signal toan image sensor 110, thereby getting video. The synchronizer 112performs this series of processing steps a number of times.

The light returning from the object is transmitted through a shootinglens 109 and then produces an image on the image sensor 110. This imagesensor 110 may be either a monochrome image sensor or a single-panelcolor image sensor with a color mosaic. The video signal of the capturedimage is transmitted through the video signal line 111 to reach theimage processor 108.

In this embodiment, the polarized light source section 120 shown in FIG.1A is realized by the light source 104, the light guide 105, the planeof polarization control element 106 and the illuminating lens 107.Meanwhile, the image capturing section 140 shown in FIG. 1A is realizedby the shooting lens 109 and the image sensor 110. And the varyingintensity processing section 160 and the normal estimating section 170shown in FIG. 1A are realized by the image processor 108.

Next, it will be described with reference to FIG. 2 how the plane ofpolarization control element 106 operates.

First, second, third and fourth images are captured in respective states203, 204, 205 and 206 in which the plane of polarization has an angle of0, 45, 90 and 135 degrees, respectively. These angles do not always haveto be increased on a 45 degree basis. But the angle of increment mayalso be any other value obtained by dividing 180 degrees by an integerof three or more. If the image sensor has high sensitivity or if theillumination has high illuminance, then the exposure process time can beshortened. As a result, the angle of rotation can be set more finely.

According to the documents described above, the time it takes to rotatethe plane of polarization may be as long as approximately 20 ms when theoperating speed is low but may also be as short as 40 to 100 M sec whenthe operating speed is high. If a high-response-speed liquid crystalmaterial is used and if the sensitivity of the image sensor is increasedto a level that is high enough to get an image captured in such a shorttime, performance that is high enough to shoot a moving picture can bemaintained even when the plane of polarization is rotated to those fourdirections one after another during shooting.

As can be seen easily from FIG. 1C, the optical axis of the illuminatinglens 107 is substantially aligned with that of the shooting lens 109.This arrangement is adopted in order to avoid casting shadows on theobject as perfectly as possible when the object is monitored with anendoscope.

It should be noted that when an endoscope is used normally, the objectcan be irradiated with non-polarized light in many cases. According tothe present invention, by adding together mutually different images asthe first through fourth images, for example, a non-polarized averagelight intensity image can be generated. The present inventors discoveredvia experiments that when the images represented by multiple polarizedlight rays, of which the planes of polarization were defined by anglesψI at regular intervals and which had been radiated toward, and hadreturned from, the object were added together, the effect ofpolarization was canceled and the effect eventually achieved was thesame as the one achieved by using a non-polarized light source.

FIG. 3 shows how the plane of polarization of polarized light source hasits angle ψI defined. As described above, an X-Y coordinate system isdefined with respect to the object. In this case, the angle ψI isdefined to be positive in the positive Y-axis direction with thenegative X-axis direction set to be 0 degrees. If the angle ψI is savedfor reflected light, then the respective planes of polarization of thereflected light and the incident light will have the same angle. And ifthe angle ψI of the plane of polarization is going to be increased ordecreased, the same polarization state will recur over and over again ina period of 180 degrees. That is to say, a function that uses the angleof the plane of polarization as a variable is a periodic function thathas a period of 180 degrees. In this description, the angle ψI of theplane of polarization of polarized light source will be sometimesreferred to herein as an “incident plane of polarization angle”.

FIGS. 4( a) and 4(b) illustrate an exemplary arrangement for the imagecapturing plane of the image sensor 110. As shown in FIG. 4( a), anumber of photosensitive cells (i.e., photodiodes) are arranged incolumns and rows on the image capturing plane. When a color image isgoing to be captured, color mosaic filters, which transmit light rayswith three different wavelengths associated with RGB, are arranged asshown in FIG. 4( b). Each of these photosensitive cells generates, byphotoelectric conversion, an electrical signal representing the quantityof the light received. In this manner, a conventional image sensor tocapture a light intensity image may be used as the image sensor 110. Inthis embodiment, if the illumination is plane polarized light, an imageis captured with its plane of polarization rotated, thereby obtaininginformation about the object's surface.

As a method for getting a color image and a polarized image at the sametime, not only the method in which the acquisition of the polarizedimage is developed on the time axis as is done in the present inventionbut also a so-called “color plane sequential” method in which the colorimage capturing is developed on the time axis could be adopted as well.

FIG. 5 illustrates an alternative configuration that may be used forsuch a purpose. Unlike the configuration shown in FIG. 1C, thenon-polarized white light emitted from the light source 104 is changedby a rotating color filter 501 into color light, which is sequentiallyradiated over a period of time. The light guide 105 transmits the colorlight as it is. The light returning from the object is usuallypolarized. However, an image sensor to capture a monochrome image may beused. For example, a monochrome polarized image sensor that uses apatterned polarizer as disclosed in Patent Document No. 4 may be used.

FIG. 6 illustrates an example of a monochrome polarized image sensor inwhich such a patterned polarizer is arranged. An image sensor like thismay be implemented as a polarization imaging element that uses aphotonic crystal as disclosed in Patent Document No. 4.

The pattern polarizer of this monochrome polarized image sensor haswavelength dependence, and therefore, cannot obtain a polarized image inthe entire wavelength range that covers all of the RGB ranges. Forexample, if the patterned polarizer is designed adaptively to the Bwavelength range, then only a B polarized image can be obtained.Furthermore, to obtain a polarized image (and the degree of polarizationand the polarization angle, among other things), a sort of spatialdifference processing should be done on a 2×2 cell using a spatiallyprocessed image of the polarization mosaic. As a result, moire will beinevitably produced on the B polarized image due to the influence ofthat difference processing. Unlike the moire pattern that has beenproduced through mere pixel sampling, the moire pattern produced byusing the pattern polarizer has been caused mainly through the spatialimage processing to obtain the polarized image as described above. Andthe present inventors discovered and confirmed via experiments that themoire pattern produced by using the pattern polarizer was much morenoticeable than the moire pattern produced through normal pixelsampling.

Consequently, with the configuration shown in FIG. 5 adopted, the imagequality would be debased significantly when a polarized image ismonitored.

On the other hand, according to this embodiment, polarizationinformation can be obtained on a pixel-by-pixel basis using a normalimage sensor, and therefore, such a problem can be avoided. That is tosay, the image capturing section of this embodiment receives thereturning light directly (i.e., by way of no polarizers) and outputs asignal representing an intensity value. On top of that, polarized imagescan also be obtained for respective wavelength components associatedwith the three primary colors of RGB, and a polarized image sensor,which would raise the overall cost when required, is no longernecessary, either.

Next, it will be described how the intensity varies when the plane ofpolarization of polarized light is rotated. In the following example,the object is not the mucosa of an organism's organ but an object madeof a general material such as plastic or wood as an example. Thisexample is taken because light is basically specular-reflected from thesurface of the mucosa and because reflection can be regarded as the samephysical phenomenon irrespective of the material of the object.

FIGS. 7A and 7B illustrate polarized images of a ceramic cup with asmooth surface and a wood plate with microfacet that were captured asobjects by the present inventors. Specifically, the two images on theleft-hand side of FIG. 7A are light intensity images that are obtainedwhen the object is illuminated with polarized light with an incidentplane of polarization angle ψI of 0 degrees. On the other hand, the twoimages on the right-hand-side of FIG. 7A are light intensity images thatare obtained when the object is illuminated with polarized light with anincident plane of polarization angle ψI of 90 degrees.

Meanwhile, the four images shown in FIG. 7B are schematicrepresentations of the four images shown in FIG. 7A. As can be easilyfrom the images shown at the top of FIGS. 7A and 7B, even if thepolarization plane of the polarized light was changed, no significantvariation was observed in the intensity pattern of the ceramic cup withthe smooth surface. As for the wood plate with a lot of microfacets, onthe other hand, it turned out that when the angle ψI of the polarizationplane of the polarized light was changed, a significant variationoccurred in the light intensity image observed as can be seen easilyfrom the images shown at the bottom of FIGS. 7A and 7B. Such adifference can be explained as follows.

FIG. 8 illustrates how polarized light is incident on the surface 801 atan angle of incidence that is close to zero degrees and how the specularreflected light is observed with a camera. The respective angles definedby the polarization planes of the incident polarized light are differentfrom each other by 90 degrees between portions (a) and (b) of FIG. 8.However, even though the reflected plane polarized light travels in adifferent direction from the incident light, the intensity (i.e., theenergy) of the reflected light is almost the same as that of theincident light for the following reasons:

FIG. 9 is a graph showing the dependence of the specular reflectanceaccording to the Fresnel theory on the angle of incidence. In FIG. 9,the abscissa represents the angle of incidence and the ordinaterepresents the Fresnel reflectance. These dependence curves are drawn onthe supposition that the refractive index NN is 1.8. The angles ofincidence of around 0 through around 15 degrees, which can be regardedas representing substantially perpendicular incidence, fall within therange 901. As can be seen from this graph, both P and S waves havesubstantially the same reflectance in this range 901. Therefore, if thepolarized light is incident substantially perpendicularly onto thesurface, then it makes almost no difference for the surface and thelight is reflected in the same behavior, no matter whether the polarizedlight is actually a P-wave or an S-wave. This fact is satisfiedextensively by any natural object with a refractive index n of 1.4 to2.0.

As described above, if polarized light is incident on a smooth surfaceat an angle of incidence of almost zero degrees, reflected once and thenobserved, the energy of the reflected light does not change, and theintensity Y observed does not change, either, even when the plane ofpolarization of the polarized light is rotated by ψI degrees.

FIG. 10A is a graph showing how the intensity value of the same pixelvaried in a situation where a light intensity image was shot with theplane of polarization of the polarized light, impinging on the surfaceof a wood plate, changed. FIG. 10B is a light intensity image of thatwood plate as the object of shooting (i.e., a light intensity imageunder non-polarized light). And FIG. 10C schematically illustrates thesurface microfacets of the wood plate shown in FIG. 10B.

FIG. 11 shows the behavior of the intensities Y of a particular pixel ofa light intensity image that were obtained when the plane ofpolarization of the polarized light had angles ψI of 0, 45, 90 and 135degrees, respectively. As can be seen from this graph, the intensity Yvaried periodically according to the angle ψI of the plane ofpolarization of each polarized light. The surface of the wood plate isnot smooth but has a lot of grooves, where the incident light producesinterreflection. Consequently, the intensity Y would vary according tothe angle ψI of polarization of the light.

The reason will be described in detail below.

FIG. 12 illustrates how a groove 1201 that has been formed on a surfaceproduces interreflection twice on its slopes. That kind ofinterreflection would be produced on an uneven surface of variousnatural objects including cloth, wood, human skin and leather. In thiscase, the properties of reflections are important the first and secondtimes around, but the interreflection is almost negligible for the thirdtime and on, because the intensity is small. Thus, in this example, onlya situation where the interreflection occurs twice will be described.Generally speaking, if the properties of reflection are roughlyclassified into specular reflection and diffuse reflection, there ariseone of the following four situations:

-   -   1) diffuse reflection the 1^(st) time around and specular        reflection the 2^(nd) time around;    -   2) diffuse reflection both of the 1^(st) and 2^(nd) times        around;    -   3) specular reflection the 1^(st) time around and diffuse        reflection the 2^(nd) time around; and    -   4) specular reflection both of the 1^(st) and 2^(nd) times        around.

Among these four situations, in situations 1) and 2), when the light isdiffuse-reflected the first time around, the diffuse-reflected lightgets non-polarized and is reflected in every direction. However, theresults of experiments revealed that when the object was colored and hadlow intensity, the diffuse reflection component of this first timearound was a minor one, which means that a relatively small quantity oflight penetrated the object. Rather, the specular reflection insituations 3) and 4), which is complementary to situation 1), prevailover the diffuse reflection according to Fresnel theory. Meanwhile, ifdiffuse reflection is produced the second time around as in situation3), it can be seen easily, considering the geometric relation betweenthe incident and reflected light rays, that situation 3) involvessituation 4). In that case, no matter whether the degree of polarizationor intensity is used as a reference, the major intensity component willbe produced by specular reflection.

Consequently, situation 4), in which specular reflection is producedboth of the first and second times around, may be regarded as thedominant phenomenon. If the tilted surface in the groove is not quitesmooth and if the illumination is not quite parallel light, evenspecular reflected light is not ideal one. That is why the presentinventors confirmed via experiments that even if the specular reflectioncondition was not satisfied completely, these two reflections could beobserved, and the image could be captured, relatively easily and thepolarization property was caused by the specular reflection.

Next, look at portions (a) and (b) of FIG. 12, in which illustrated is aportion of a groove 1201 on the surface of an object. On the object'ssurface, at least a portion of the groove 1201 runs in one direction,which will be referred to herein as the “main axis direction”. Actually,however, the groove 1201 does not have to run linearly but may be acurved one, too. Even in such a curved groove, its portion can also beapproximated to be a linear groove that runs in the main axis direction.

It should be noted that a cross section of the groove 1201 on theobject's surface can be approximated to be a V-shape. That is why agroove on the surface of an organism's organ can be called a “V-groove”.However, a cross section of such a V-groove does not have to have anexactly V-shape but may have a curved portion, too. In any case, as longas there is a groove with a generally “V-shaped” cross section on theobject's surface, the following description is applicable. Likewise, astructure in which a recess interposed between two adjacent raisedportions runs in the direction coming out of the paper such as the oneto be described later with reference to FIG. 22 is another example of aV-groove.

As shown in portion (a) of FIG. 12, polarized light incidentperpendicularly to the main axis direction 1202 of the groove is aP-wave. Look at FIG. 9 again, and it can be seen that if the object'sgroove 1201 has a tilt angle of approximately 45 degrees and if light isincident from right over the groove 1201, the reflectance of a P-wavebecomes much lower than that of an S-wave in the range 902 of that angleof incidence as can be seen from the graph showing the Fresnelreflectance. The reflectance of the P-wave further decreases as theP-wave goes through reflection first and second times around. On theother hand, the S-polarized light shown in portion (b) of FIG. 12 doesnot have its reflectance decreased so much even after having gonethrough the reflections first and second times around. As a result, onthe plane of polarization of the P-wave that has been incident on thegroove, the reflected light comes to have very low energy and decreasedintensity. On the other hand, on the incident plane of polarization ofthe S-wave, the reflected light has not had its energy attenuated somuch and still maintains high intensity.

If the surface groove is supposed to be as such, the variation in theintensity of the reflected light that was caused by rotating the planeof polarization of the incident light in an experiment can be accountedfor.

The present inventors discovered that the function representing thevariation in intensity Y that was caused by getting the polarized lightreflected twice from the groove changed in substantially the same way asin a situation where non-polarized light was incident there.Hereinafter, this respect will be described.

FIG. 13( a) illustrates a groove on the object's surface as viewed fromright over that surface, which corresponds to looking down on FIG. 12from right over the paper. In FIG. 13( a), shown are X and Y coordinateson a plane that is parallel to the image capturing plane. Also shown isthe angle ψ formed between the direction that intersects at right angleswith the main axis direction 1202 of the groove 1201 and the positiveX-axis direction. FIG. 13( b) shows the angle ψI of the plane ofpolarization of the polarized light that has been incident on theobject. And FIG. 13( c) illustrates, in combination, what is illustratedin FIGS. 13( a) and 13(b) in the same drawing. In the followingdescription, the direction of the groove will be specified herein by theangle ψ, which is different from the azimuth angle of the groove's mainaxis by 90 degrees.

FIG. 14A illustrates the energies of incident light to be distributedperpendicularly and horizontally with respect to a groove in a situationwhere the plane of polarization is supposed to be aligned with theX-axis for the sake of simplicity (i.e., ψI=0). The direction of thegroove is specified by the angle ψ. Suppose the incident light isreflected twice in the groove as shown in FIG. 12. In that case, theintensity of plane polarized light, of which the plane of polarizationhas a certain angle φ, is measured. FIG. 14B shows the angle ψ of theplane polarized light, of which the intensity is measured. If thepolarized light intensity at the angle φ is represented by I (ψ, φ),then the intensity can be represented by the following Equation (1)where the energy reflectances in the groove direction (ψ) and the mainaxis direction (πL/2−ψ) are identified by A and B, respectively:

I(ψ,φ)=A cos² ψ cos² (ψ−φ)+B sin² ψ sin² (ψ−φ)   (1)

By modifying Equation (1), this polarized light intensity I (ψ, φ) canbe represented by the following Equation (2):

$\begin{matrix}{{I( {\psi,\phi} )} = {\frac{( {A + B} )}{4} + {\frac{( {A - B} )}{4}\cos \; 2\psi} + {\lbrack {\frac{( {A + B} )}{8} + {\frac{( {A - B} )}{4}\cos \; 2\psi} + {\frac{( {A + B} )}{8}\cos \; 4\psi}} \rbrack \cos \; 2\phi} + {\lbrack {{\frac{( {A - B} )}{4}\sin \; 2\; \psi} + {\frac{( {A + B} )}{8}\sin \; 4\psi}} \rbrack \sin \; 2\phi}}} & (2)\end{matrix}$

As can be seen from this Equation (2), the polarized light intensity I(ψ, φ) varies in the period π with respect to φ.

Suppose the incident plane of polarization angle is the general value ψIinstead of 0 degrees. In that case, it can be seen, from the foregoingdiscussion, that the polarized light intensity in a situation where theincident plane of polarization angle is φI and the viewing angle is φ isgiven by the following Equation (3):

$\begin{matrix}{{I( {{\psi - \psi_{I}},{\phi - \psi_{I}}} )} = {{\frac{( {A + B} )}{4} + {\frac{( {A - B} )}{4}\cos \; 2( {\psi - \psi_{I}} )}}+={{\lbrack {\frac{( {A + B} )}{8} + {\frac{( {A - B} )}{4}\cos \; 2( {\psi - \psi_{I}} )} + {\frac{( {A + B} )}{8}\cos \; 4( {\psi - \psi_{I}} )}} \rbrack \cos \; 2( {\phi - \psi_{I}} )} + {\quad{\lbrack {{\frac{( {A - B} )}{4}\sin \; 2( {\psi - \psi_{I}} )} + {\frac{( {A + B} )}{8}\sin \; 4( {\psi - \psi_{I}} )}} \rbrack \sin \; 2( {\phi - \psi_{I}} )}}}}} & (3)\end{matrix}$

The polarized light intensity given by this Equation (3) is measured ata viewing angle φin a particular direction. That is why in measuring theaverage intensity of non-polarized light, the polarized light intensityrepresented by Equation (3) needs to be integrated for one period withrespect to the viewing angle φ. In this case, one period is 180degrees=π. As a result of this integral operation, the sine and cosinefunction with respect to φ become equal to zero. That is to say, thelight intensity PY (ψI, φ) to be measured in a situation where polarizedlight with an incident plane of polarization angle ψI is incident on agroove that is specified by an angle ψ and then reflected twice can berepresented as a periodic function of 180 degrees with respect to 104 Ias in the following Equation (4):

$\begin{matrix}\begin{matrix}{{{PY}( {\psi_{I},\psi} )} = {\int_{0}^{\pi}{{I( {{\psi - \psi_{I}},{\phi - \psi_{I}}} )}{\phi}}}} \\{= {\frac{A + B}{4} + {\frac{A - B}{4}\cos \; 2( {\psi - \psi_{I}} )}}}\end{matrix} & (4)\end{matrix}$

If the light intensity PY (ψI, φ) becomes a cosine function of ψI asrepresented by this Equation (4), the light intensity PY (ψI, φ) comesto have a maximum value when ψ=ψI. That is why such an angle ψ=ψI atwhich the light intensity PY (ψI, φ) has a maximum value will bereferred to herein as an “intensity maximizing angle YPH”. As for theamplitude of variation, considering that the cosine function term varieswithin the range of +1 through −, the degree of modulation of theintensity variation can be considered. And that ratio will be referredto herein as a “degree of light intensity modulation YD”, which can becalculated by the following Equation (5):

$\begin{matrix}{{YD} = {\frac{{MAX} - {MIN}}{{MAX} + {MIN}} = \frac{B - A}{A + B}}} & (5)\end{matrix}$

It should be noted that the intensity maximizing angle YPH and thedegree of intensity modulation YD are given on a pixel-by-pixel basis.That is why an image, in which the intensity maximizing angle YPH is setfor each of the constituent pixels thereof, will be referred to hereinas an “intensity maximizing angle image YPH”. Likewise, an image, inwhich the degree of intensity modulation YD is set for each of theconstituent pixels thereof, will be referred to herein as a “degree ofintensity modulation image YD”.

In this case, the intensity maximizing angle YPH and the degree ofintensity modulation YD are quantities corresponding to a polarizationmain axis angle and a degree of polarization, respectively, in normalpolarized light measuring. However, their quantitative relation has notbeen defined clearly yet. Thus, in order to clarify their relation, letus consider what polarization state twice-reflected light will have in asituation where non-polarized light has been incident on a groove.

FIG. 15 illustrates a situation where non-polarized light 1501 has beenincident on a groove. If the non-polarized light 1501 has been incidenton a groove that has an angle ψ, then its energy would be equallydistributed to the main axis direction of the groove and the directionthat intersects with the former direction at right angles. That is whyenergies multiplied by their energy reflectances A and B would beemitted in that groove's main axis direction and the direction thatintersects with the main axis direction at right angles, respectively.If the polarized light is measured at an angle φ, the polarized lightintensity will have its maximum value (at the reflectance B) in thegroove's main axis direction and have its minimum value (at thereflectance A) in that direction that intersects with the main axisdirection at right angles as already described with reference to FIG.12. The degree of polarization DOP is calculated by the followingEquation (6):

$\begin{matrix}{{D\; O\; P} = {\frac{{MAX} - {MIN}}{{MAX} + {MIN}} = \frac{B - A}{A + B}}} & (6)\end{matrix}$

As can be seen from the foregoing discussion, it turned out that theintensity maximizing angle YPH, which is the phase angle of the lightintensity variation in a situation where the angle ψI of the plane ofpolarization of polarized light has been rotated, agrees with thepolarization main axis when non-polarized light is radiated. Likewise,it also turned out that the degree of intensity modulation YD, which isthe amplitude of the light intensity variation in a situation where theangle ψI of the plane of polarization of polarized light has beenrotated, agrees with the degree of polarization DOP when non-polarizedlight is radiated. Consequently, the Fresnel reflection theory and thesurface normal discussion on the supposition that non-polarized light isradiated can be used to analyze a variation in polarized light intensityaccording to the present invention.

The image processing processor 108 of this embodiment gets the intensitymaximizing angle image YPH and the degree of intensity modulation imageYD as described above, thereby obtaining information about the object'ssurface microfacets. Hereinafter, it will be described with reference toFIG. 16 what configuration the image processing processor 108 may haveand how the processor 108 may operate.

FIG. 16 is a block diagram illustrating a configuration for the imageprocessing processor 108. By changing the incident plane of polarizationangles ψI of the light from 0 degrees into 45, 90 and 135 degrees andilluminating the object with polarized light at each of these incidentplane of polarization angles ψI, a group 1601 of four light intensityimages is obtained. Then, the group 1601 of four light intensity imagesthus captured is input to the varying intensity processing section 160of this image processing processor 108.

As described above, the intensity variation in a situation where theplane of polarization of polarized light is rotated becomes a cosinefunction with a period of 180 degrees. The varying intensity processingsection 160 fits the intensity variation to the cosine function. Y(ψI)representing the intensity variation can be given by the followingEquation (7) using the angle ψI of the plane of polarization of thelight as a variable:

Y(ψ₁)=Y _(ψ1) _(—) _(ave) +A ₁ cos(2(ψ₁−ψ₀))   (7)

FIG. 17 shows the cosine function of this intensity variation andindicates the meanings of the amplitude AI, the phase ψo and the averageY_(ψI) _(—) _(ave) described above. The four sample points are plottedright on that cosine function for the sake of simplicity.

These values can be estimated by fitting the cosine function based onthe four angular samples that have been obtained at regular intervals inthe following manner. First of all, the intensity Y_(ψI) _(—) _(ave) ofthe original image under non-polarized light is calculated by thefollowing Equation (8). The right side of this Equation (8) means addingtogether, and calculating the average of, four light intensity imagesthat have been obtained from an object that is illuminated withpolarized light at angles ψI of 0, 45, 90 and 135 degrees, respectively.The intensity Y_(ψI) _(—) _(ave) approximately reproduces a lightintensity image under non-polarized light and can be used as a normallyobserved image for an endoscope. That is why the intensity Y_(ψI) _(—)_(ave) can be referred to herein as a “non-polarized light average lightintensity image”.

$\begin{matrix}\begin{matrix}{Y_{\psi {I\_ AVE}} = {\frac{1}{4}( {{Y( {\psi_{I} = {0{^\circ}}} )} + {Y( {\psi_{I} = {45{^\circ}}} )} + {Y( {\psi_{1} = {90{^\circ}}} )} + {Y( {\psi_{I} = {135{^\circ}}} )}} )}} \\{\approx {\frac{1}{2}( {Y_{\max} + Y_{\min}} )}}\end{matrix} & (8)\end{matrix}$

Next, optimum fitting from the sampled intensities to the cosinefunction is carried out using a minimum mean square error. In this case,the optimum fitting process is begun by carrying out sampling in thefour directions that are defined by 0, 45, 90 and 135 degrees,respectively. Since the cosine function is determined by the three kindsof information that are amplitude, phase and average, the number ofsamples for use to determine the cosine function does not have to befour but may actually be any other number as long as the number is atleast three. Nevertheless, if samples are taken at a regular interval of45 degrees in this manner, the optimum fitting can be simplified.

First of all, the square error E of the intensities at the polarizationangles of 0, 45 (=π/4), 90 (=π/2) and 135 (=π/4) degrees is defined bythe following Equation (9):

$\begin{matrix}{E = {{( {{Y( {\psi_{I} = 0} )} - I_{0}} )^{2} + ( {{Y( {\psi_{I} = \frac{\pi}{4}} )} - I_{1}} )^{2} + ( {{Y( {\psi_{I} = \frac{\pi}{2}} )} - I_{2}} )^{2} + ( {{Y( {\psi_{I} = \frac{3\pi}{4}} )} - I_{3}} )^{2}} = {( {Y_{\psi {I\_ AVE}} + {A_{I}{\cos ( {2\psi_{O}} )}} - I_{0}} )^{2} + ( {Y_{\psi {I\_ AVE}} + {A_{I}{\sin ( {2\psi_{O}} )}} - I_{1}} )^{2} + ( {Y_{\psi {I\_ AVE}} - {A_{I}{\cos ( {2\psi_{O}} )}} - I_{2}} )^{2} + ( {Y_{\psi {I\_ AVE}} - {A_{I}{\sin ( {2\psi_{O}} )}} - I_{3}} )^{2}}}} & (9)\end{matrix}$

The phase ψo of the cosine function that minimizes this square error canbe calculated by the following Equation (10):

$\begin{matrix}{\frac{\partial E}{\partial\psi_{O}} = {{4{A_{I}\lbrack {{( {I_{3} - I_{1}} ){\cos ( {2\psi_{O}} )}} + {( {I_{0} - I_{2}} ){\sin ( {2\psi_{O}} )}}} \rbrack}} = 0}} & (10)\end{matrix}$

Based on this equation, the solutions can be given by the followingEquations (11) and (12):

$\begin{matrix}\{ \begin{matrix}{\psi_{O}^{( + )} = {\frac{1}{2}{\cos^{- 1}( \sqrt{\frac{c^{2}}{a^{2} + c^{2}}} )}}} \\{\psi_{O}^{( - )} = {\frac{1}{2}{\cos^{- 1}( {- \sqrt{\frac{c^{2}}{a^{2} + c^{2}}}} )}}}\end{matrix}  & (11) \\\{ \begin{matrix}{a \equiv ( {I_{3} - I_{1}} )} \\{c \equiv ( {I_{0} - I_{2}} )}\end{matrix}  & (12)\end{matrix}$

A mathematical function such as an inverse trigonometric functiongenerally imposes the following constraint:

0≦a cos(x)≦π  (13)

Considering this angular range, by making classification based on themagnitudes of a and c, the respective angles at which the maximum andminimum values are obtained can be calculated by the following Equations(14):

$\begin{matrix}\{ \begin{matrix}{{{{if}\mspace{14mu} a} < {0\mspace{14mu} {and}\mspace{14mu} c} > 0},} & {\psi_{Omin} = {\frac{\pi}{2} + \psi_{O}^{( + )}}} & {\psi_{Omax} = \psi_{O}^{( + )}} \\{{{{if}\mspace{14mu} a} < {0\mspace{14mu} {and}\mspace{14mu} c} < 0},} & {\psi_{Omin} = {\frac{\pi}{2} + \psi_{O}^{( - )}}} & {\psi_{Omax} = \psi_{O}^{( - )}} \\{{{{if}\mspace{14mu} a} > {0\mspace{14mu} {and}\mspace{14mu} c} < 0},} & {\psi_{Omin} = \psi_{O}^{( + )}} & {\psi_{Omax} = {\frac{\pi}{2} + \psi_{O}^{( + )}}} \\{{{{if}\mspace{14mu} a} > {0\mspace{14mu} {and}\mspace{14mu} c} > 0},} & {\psi_{Omin} = \psi_{O}^{( - )}} & {\psi_{Omax} = {\frac{\pi}{2} + \psi_{O}^{( - )}}}\end{matrix}  & (14)\end{matrix}$

The ψ 0max value at which the maximum value is obtained can be used asit is as the intensity maximizing angle image YPH:

YPH=ψ_(Omax)   (15)

Next, the maximum and minimum values of the amplitude are obtained.First of all, to obtain the amplitude A_(I), the square error isminimized by the following Equations (16) and (17):

$\begin{matrix}{\frac{\partial E}{\partial A_{I}} = 0} & (16) \\{A_{I} = {\frac{1}{2}\lbrack {{( {I_{0} - I_{2}} ){\cos ( {2\psi_{O}} )}} - {( {I_{3} - I_{1}} ){\sin ( {2\psi_{O}} )}}} \rbrack}} & (17)\end{matrix}$

Using the amplitude A_(I), the maximum and minimum values of theamplitude are calculated by the following Equations (18):

Y _(max) =Y _(ψI) _(—) _(AVE) +A _(I)

Y _(min) =Y _(ψI) _(—) _(AVE) −A _(I)   (18)

Thus, if the maximum and minimum values Ymax and Ymin of the amplitudegiven by these Equations (18) are applied as MAX and MIN to Equation(5), the degree of intensity modulation image YD can be obtained.

Normal optimum fitting to a cosine function can be carried out on threeor more samples and its method is disclosed in Patent Document No. 5,for example.

By performing these processing steps, the intensity maximizing angleimage YPH and the degree of intensity modulation image YD can beobtained. In FIG. 16, the intensity maximizing angle image YPH and thedegree of intensity modulation image YD are identified by the referencesigns YPH and YPH, respectively. As shown in FIG. 16, the intensitymaximizing angle image YPH and the degree of intensity modulation imageYD are passed to an azimuth angle processing section 1604 and a zenithangle processing section 1606, respectively.

FIG. 18A shows the azimuth angle and zenith angle that are two anglesthat determine the direction of a normal to the object's surface. Thenormal vector is a three-dimensional vector but its length has beennormalized to one. Thus, the normal vector has a degree of freedom oftwo. And when represented as an angle, the normal vector can berepresented by the azimuth angle ψ within the screen and the zenithangle θ with respect to the line of sight. In a normal right-handedsystem, X and Y axes are defined within the image and the directionindicated by the Z-axis becomes the line of sight (i.e., optical axis)direction. The relation between the normal and three components (Nx, Ny,Nz) is as shown in FIG. 18A. Once the azimuth angle ψ and the zenithangle θ have been obtained, the surface normal at that point isrepresented by the following Equations (19):

N_(X)=cos ψ sin θ

N_(Y)=sin ψ sin θ

N_(Z)=cos θ  (19)

The azimuth angle processing section 1604 calculates the azimuth angle ψusing the intensity maximizing angle image YPH. In this case, based onthe conclusion of the foregoing discussion, the Fresnel theory aboutspecular reflection in a situation where non-polarized light has beenincident is used as a reference. According to that theory, as disclosedin Non-Patent Document No. 2, non-polarized light is incident and issubjected to polarimetry by a polarizer that is arranged in front of thecamera. In that case, in the reflected light that has beenspecular-reflected from the object's surface, the P-wave attenuates andthe S-wave prevails instead. As a result, the azimuth angle ψ of anormal to the surface of the groove becomes equal to the angle of theplane of polarization that minimizes the intensity.

The direction defined by this azimuth angle ψ agrees with the directionthat intersects at right angles with that the plane of polarization thatmaximizes the intensity. And if this relation is applied to thepolarized illumination of the present invention, the azimuth angle ψ ofa normal to the surface of the groove agrees with the direction thatintersects at right angles with the direction that maximizes thepolarized light intensity. That is to say, the azimuth angle ψ can bedetermined by the intensity maximizing angle image YPH.

In this case, the problem is that this azimuth angle ψ has an ambiguityof 180 degrees as mentioned as “180° ambiguity” in Non-Patent DocumentNo. 2. That is to say, two angles that are different from each other by180 degrees are obtained as candidates for the azimuth angle ψ of anormal to the groove's surface. To choose one of those two candidateswill be referred to herein as “ambiguity processing”.

To perform the ambiguity processing, the azimuth angle ambiguityprocessing section 1607 shown in FIG. 16 uses either a non-polarizedlight intensity image 1612 or the degree of intensity modulation imageYD.

Next, it will be described with reference to FIGS. 19A and 19Bspecifically what processing is carried out by the azimuth angleambiguity processing section 1607. FIG. 19A illustrates a lightintensity image and a cross section 1902 of a groove. Since the groovehas a bigger size than each pixel, there is an intensity distributionwithin the single groove. It should be noted that the cross section 1902of the groove has two tilted surfaces, which may have a tilt angle of 30to 60 degrees.

When the groove is irradiated with illuminating light that has come fromalmost right over the groove, the light intensity image comes to have anincreased intensity value (i.e., gets brighter) in the shadowed areas1903 and a decreased intensity value (i.e., gets darker) in the area1913 near the bottom of the groove, respectively. That is to say, theintensity values come to have a gradient inside the groove. If thegradient vectors of the intensity values are calculated by subjectingthe intensity Y_(ψI) _(—) _(ave) that has been obtained undernon-polarized light to spatial differentiation processing, intensitygradient vectors 1904 and 1905 can be obtained. These gradient vectors1904 and 1905 are calculated respectively for the two tilted surfaces inthe groove.

On the other hand, since the azimuth angle ψ obtained in the intensitymaximizing angle image YPH has an ambiguity of 180 degrees, twocandidates 1906 a and 1906 b are obtained at a point A and two morecandidates 1907 a and 1907 b are obtained at a point B. In FIG. 19A,these candidate vectors are illustrated as having slightly shifted fromthe right direction of the groove. However, this is just a schematicrepresentation that is simplified to make the angular difference to bedescribed below more easily understandable and is never an essentialone.

And if one of the two candidate azimuth angles ψ that has the greaterangular difference with respect to the intensity gradient vector isadopted, then the candidates 1906 a and 1907 a will be adopted at thepoints A and B, respectively. Consequently, the azimuth angle ψ of thecandidate 1906 a that is indicated by the downward arrow is chosen atthe point A and the azimuth angle ψ of the candidate 1907 a that isindicated by the upward arrow is chosen at the point B correctly.

Next, it will be described with reference to FIG. 19B how to choose oneof the two candidates based on the degree of intensity modulation imageYD.

If the groove is irradiated with illuminating light that has come fromalmost right over the groove, then the light will be reflected twiceinside the groove, and therefore, the degree of polarization DOP becomeshigh around the bottom of the groove. As can be seen from the theorydescribed above, even if the polarized light source is rotated, thedegree of intensity modulation value will also behave in a similarmanner. That is why the degree of intensity modulation image YD comes tohave an increased degree of intensity modulation value in the area 1908near the bottom 1913 of the groove but a decreased value outside of thegroove, respectively. If the gradient vectors of the degree of intensitymodulation image YD are calculated, degree of intensity modulationgradient vectors 1909 and 1910 can be obtained. On the other hand, sincethe azimuth angle ψ obtained in the intensity maximizing angle image YPHhas an ambiguity of 180 degrees, two candidates 1911 a and 1911 b areobtained at a point A and two more candidates 1912 a and 1912 b areobtained at a point B. And if one of the two candidate azimuth angles ψthat has the smaller angular difference is adopted by using theestimated value of the angular difference between the gradient vector ofthe degree of intensity modulation and the intensity maximizing angle,then the candidates 1911 a and 1912 a will be adopted at the points Aand B, respectively. In FIGS. 19A and 19B, the gradient vectors and thenormal vectors are illustrated as if those vectors were calculated atmutually different points. However, this illustration is adopted justfor convenience sake. In actual image processing, the gradient vectorsare calculated right at the points A and B where the normal vectors areestimated.

The foregoing description could read as stating that the normal vectorscan be obtained correctly even by using either the intensity gradientvector or the degree of intensity modulation vector by itself. However,that is a simple misunderstanding. Actually, those pieces of informationmay include a lot of noise due to spatial differentiation and aretotally unreliable by themselves. It should be noted that those piecesof information cannot be useful unless they are combined with theintensity maximizing angle image obtained by using the polarized lightsource of the present invention.

FIG. 19C is a flowchart showing the procedure of determining the azimuthangle.

First, in Step S19C01, a candidate ψ1 for the azimuth angle ψ of thegroove is obtained by rotating the (intensity maximizing angle) value ofthe intensity maximizing angle image YPH 90 degrees. Next, in StepS19C02, another azimuth angle candidate ψ2 is obtained by rotating theformer candidate ψ1 180 degrees. Then, in Step S19C03, the estimatedvalues Δ1 and Δ2 of the angular differences between the gradient angle(gradangle) that has been obtained based on either the intensitygradient or the degree of intensity modulation and the candidates ψ1 andψ2 are calculated:

Δ1=min(|ψ1−gradangle|, 360−″ψ1−gradangle|)

Δ2=min(|ψ2−gradangle|, 360−|ψ2−gradangle|)   (20)

As to whether the intensity gradient or the degree of intensitymodulation may be adopted, it may be determined by the property of theobject's surface microfacets. Actually, however, a simple situationwhere the intensity variation literally represents the surface normalrarely happens if ever. In reality, the intensity will change in variousmanners due to not just the normal but also interreflection and otherengineering effects. In comparison, the degree of intensity modulationimage can be obtained not just with good stability but also even withoutmaking any special modification to the image sensor on the sensing endif the polarized light source is improved just as disclosed in theforegoing description of the present invention. Consequently, the degreeof intensity modulation image is much more effective.

Next, in Step S19C04, a decision is made as to whether the intensitygradient or the degree of intensity modulation gradient is used. If theformer is chosen, the condition of Step S19C05 is set to choose one ofψ1 and ψ2 that has the greater angular difference, thereby determiningψ. On the other hand, if the latter is chosen, then the condition ofStep S19C06 is set to choose one of ψ1 and ψ2 that has the smallerangular difference, thereby determining ψ.

The zenith angle processing section 1606 calculates the zenith angle θusing the degree of intensity modulation image YD. In the prior art, asof now, nobody has ever accurately and theoretically clarified therelation between the degree of polarization and the zenith angle θ in asituation where the light is reflected twice by a surface groove.According to this embodiment, the Fresnel theory for a situation wherenon-polarized light is incident is adopted. In that case, as disclosedin Non-Patent Document No. 2, non-polarized light is incident andsubjected to polarimetry by a polarizer that is arranged in front of acamera. If the degree of polarization DOP of the light that has beenspecular reflected from the object's surface is calculated at this pointin time, the Fresnel theory of curves that uses a refractive index NN issatisfied between the zenith angle of the surface normal and DOP. And ifthat theory is applied to the polarized light source of the presentinvention, the zenith angle can be determined by using the value of thedegree of intensity modulation image YD instead of DOP. Nevertheless, asalso described in Non-Patent Documents Nos. 2 and 3, this zenith anglehas ambiguity that interposes a Brewster angle.

That ambiguity will be described. FIG. 20A shows how the degree ofpolarization DOP changes with the angle of incidence of the illuminationwith the refractive index NN changed from 1.5 through 3.0. The angle ofincidence becomes the zenith angle at the surface as it is. If therefractive index is increased, the peak position shifts from around 55degrees to around 70 degrees in the direction in which the angleincreases. If non-polarized light is incident on the object's surface atan angle of incidence θ and then specular reflected from the surface atthe same angle of emittance θ, then the degree of polarization DOP canbe calculated by the following Fresnel reflection equations using therefractive index NN of the material:

$\begin{matrix}{{{D\; O\; P} = \frac{{FF}_{S} - {FF}_{P}}{{FF}_{S} + {FF}_{P}}}{{FF}_{S} = ( \frac{{\cos \; \theta} - \sqrt{({NN})^{2} - {\sin^{2}\theta}}}{{\cos \; \theta} + \sqrt{({NN})^{2} - {\sin^{2}\theta}}} )^{2}}{{FF}_{P} = ( \frac{{{- ({NN})^{2}}\cos \; \theta} + \sqrt{({NN})^{2} - {\sin^{2}\theta}}}{{({NN})^{2}\cos \; \theta} + \sqrt{({NN})^{2} - {\sin^{2}\theta}}} )^{2}}} & (21)\end{matrix}$

The “ambiguity” means that when the zenith angle is calculated based onDOP, two angles will be estimated to interpose the Brewster angle, atwhich a curve reaches its peak, between them and none of them can bechosen unambiguously.

FIG. 20B shows how that ambiguity interposing the Brewster angle can besolved. The groove at an organism's surface mucosa to be observed withan endoscope is supposed to have a tilt angle that is equal to orsmaller than 55 degrees, which substantially corresponds to the Brewsterangle. And by defining the search range from 0 degrees through theBrewster angle as shown in FIG. 20B, the difficulty is overcome.

FIG. 20C is a flowchart showing how to determine the zenith angle.

First, in Step S20C01, the maximum value MINDIF of the difference indegree of polarization is set. Next, in Step S20C02, the Brewster angleθ_(B) is obtained theoretically based on the refractive index NN. Then,θ is set in Step S20C03 to be zero degrees. And if it turns out in thenext processing step S20C04 that θ is smaller than the Brewster angle,calculations are made in the processing steps that follow. Specifically,first, in Step S20C05, the theoretical degree of polarization DOP givenby Equation (21) is obtained. Next, in Step S20C06, the absolute valueof the difference DIF between that DOP value and the degree of intensitymodulation YD is calculated. If DIF turns out to be smaller than MINDIFin the next processing step S20C07, θ is set to be θ_(MIN) and DIF isset to be MINDIF. After that, in Step S20C09, that θ angle is increasedby one degree at a time to continue the same loop processing. And ifthat loop processing comes to an end in Step S20C04, then θ_(MIN) isdetermined to be the zenith angle.

The normal image generating section 1608 obtains a normal vector (Nx,Ny, Nz) with respect to an object's surface in the camera coordinatesystem by Equation (19) using the azimuth angle φ and zenith angle θthus obtained, and defines that normal vector as representing atwo-dimensional normal image.

The light intensity image generating section 1609 generates a lightintensity image based on a physical reflection modeling formula bygiving the camera viewpoint direction and the illuminating light sourcedirection to the normal image thus obtained. In this example, theCook-Torrance model is used as a modeling formula that represents theobject's specular reflection well. According to the Cook-Torrance model,the intensity Is can be represented by the following Equation (22):

$\begin{matrix}{I_{S} = {K\frac{{FG}\frac{1}{4m^{2}\cos^{4}\alpha}{\exp( {- \frac{\tan^{2}\alpha}{m^{2}}} )}}{\cos \; \theta_{r}}}} & (22)\end{matrix}$

FIG. 18B shows a relation between the vector and the angle in asituation where the Cook-Torrance model is used and illustrates asurface normal N, a light source vector L and a viewpoint vector V. If abisector vector H of the light source vector L and the viewpoint vectorV is used, α in the Equation (22) is the angle defined by the bisectorvector H with respect to the normal N and θr is the angle formed betweenthe viewpoint vector V and the normal N. The Fresnel coefficient F andthe geometric attenuation factor G are represented by the followingEquations (23) and (24), respectively:

$\begin{matrix}{F = {\frac{1}{2}\frac{( {g - c} )^{2}}{( {g + c} )^{2}}( {1 + \frac{\lbrack {{c( {g + c} )} - 1} \rbrack^{2}}{\lbrack {{c( {g - c} )} + 1} \rbrack^{2}}} )}} & (23) \\{G = {\min \{ {1,\frac{2( {N \cdot H} )( {N \cdot V} )}{( {V \cdot H} )},\frac{2( {N \cdot H} )( {N \cdot L} )}{( {V \cdot H} )}} \}}} & (24)\end{matrix}$

Also, the coefficient K is a coefficient concerning the illuminance ofthe incoming light. If this Cook-Torrance model is used, the lightintensity image can be generated based on the surface normal image. Forthat purpose, however, not only the refractive index NN but also theviewpoint vector V, the light source vector L and other geometricsettings need to be determined as well.

The illuminating direction setting section 1610 is a component thatdetermines this light source vector, which may be set arbitrarily by adoctor, who is an observing user when making an endoscope diagnosis.

By performing these processing steps, surface microfacets are estimatedbased on a two-dimensional image, thereby generating a light intensityimage 1611 in which the surface microfacets are reflected as a surfacenormal image on the light intensity. Since this is an image that isbased on the estimated normal image, the illumination can be changedfreely on the computer. As a result, it is possible to overcome one ofthe problems with an endoscope that it is difficult to observe surfacemicrofacets because the position of the light source cannot be changed.

It should be noted that each of the components shown in FIG. 16 could beimplemented as either a piece of special-purpose hardware or acombination of hardware and software. In this embodiment, the functionsof at least some of the components described above may be performed by aprocessor. Also, the operation of the image processing section of thisembodiment can be defined by a computer program that is stored in amemory in the image processing section. Such a computer program is a setof instructions that make the processor provided for the imageprocessing apparatus perform the various processing steps (shown inFIGS. 19C and 20C).

Portion (a) of FIG. 21 illustrates a result of a surface normalestimation experiment according to this embodiment. The object is alenticular lens plate with a semicylindrical cross section. To makespecular reflection prevail, the object is painted in either a red-basedcolor or a chocolate-based color. The processing described above wascarried out on such an object.

In portion (a) of FIG. 21, the upper image is the non-polarized lightintensity image 1612 shown in FIG. 16 and the lower image is the degreeof intensity modulation image YD shown in FIG. 16. In the non-polarizedlight intensity image, white (i.e., bright) pixels have highintensities. On the other hand, in the degree of intensity modulationimage, dark (i.e., low-lightness) pixels have high values (degrees ofmodulation). Portion (b) of FIG. 21 schematically illustrates only thesmall rectangular portion shown in portion (a) of FIG. 21. The lightintensity image has an iterative pattern in which high-intensityportions (i.e., white portions 2104) corresponding to convex portions2101 alternate with low-intensity portions (i.e., shadowed portions2103) corresponding to concave portions 2102. On the other hand, thedegree of intensity modulation image has an iterative pattern in whichhigh-degree-of-modulation portions (i.e., hatched portions 2105)alternate with low-degree-of-modulation portions (i.e., white portions2106). And the low-intensity portions correspond to high degrees ofintensity modulation.

FIG. 22 is a cross-sectional view of the object (i.e., the lenticularplate) shown in FIG. 21. The lenticular plate has a periodicconcavo-convex pattern. The illuminating light is reflected only oncefrom the convex portion 2201 and imaged by a camera. On the other hand,the light that produces an image of the concave portion 2201 isreflected once to have a very high intensity. Since the concave portion2202 forms a groove, the light is eventually reflected twice and thenimaged by the camera. As a result, the image produced by the light thathas been reflected from the concave portion 2202 has a relatively lowintensity.

FIG. 23 shows the profiles of the light intensity and degree ofintensity modulation shown in FIG. 21. In FIG. 23, the curve 2301represents the light intensity and the curve 2302 represents the degreeof intensity modulation. It can be seen that in a low-intensity portion(i.e., a valley of the curve 2301), the degree of intensity modulationincreases.

FIG. 24 shows plan views of another object. This object is a plasticplate through which eight grooves A through I were cut and which waspainted. Specifically, FIG. 24A is a photograph showing a lightintensity image that was obtained by shooting the real object and FIG.24B is a schematic representation thereof. Two adjacent grooves withinthis plane had an azimuth angle interval of 22.5 degrees.

FIGS. 25 and 26 respectively show the azimuth angles ψ that wereestimated by the azimuth angle ambiguity processing section 1607 shownin FIG. 16, and the zenith angles θ that were estimated by the zenithangle processing section 1606 shown in FIG. 16, with respect to therespective grooves A through I of the object shown in FIG. 24. Thehatched graphs show the correct angles and the shadows graphs show theestimated ones. The correct angles were measured with a laserdisplacement meter.

As can be seen from the results shown in FIG. 25, the azimuth anglescould be estimated with an error of approximately 10 degrees. On theother hand, as can be seen from FIG. 26, the zenith angles estimated hadmore significant errors.

FIG. 27 shows light intensity images that were obtained by performingevery processing shown in FIG. 16 on the object shown in FIG. 24.Specifically, portions (a), (b), (c) and (d) of FIG. 27A show the imagesthat were generated by illuminating the object with the light sourcethat was located on the right-hand side of the screen, on the left-handside of the screen, under the screen, and over the screen, respectively.FIG. 27B illustrates schematic representations thereof and its portions(a) through (d) exactly correspond to those portions (a) through (d) ofFIG. 27A. It can be seen that the grooves identified by A through Icould be represented clearly by changing the illuminating direction.Consequently, a definitely embossed image can be provided for theendoscope image that is illuminated with only the light source locatedright in front of the object as shown in FIG. 24A.

Embodiment 2

Next, a second embodiment of an image processing apparatus according tothe present invention will be described with reference to FIG. 28.

The image processing apparatus of this embodiment has almost the sameconfiguration as its counterpart shown in FIG. 1C but has an additionalcolor filter 2801, which is the only difference from the apparatus shownin FIG. 1C.

The present inventors discovered and confirmed via experiments that thepolarization state observed at a groove changes significantly with thecolor of the object. Specifically, when shot within a narrow range ofaround a wavelength of 520 nm, the image of a chocolate or red objectbecomes generally dark, and very intense polarization can be observed atthe groove. On the other hand, when the object has a yellow-based color,the image shot is bright as a whole but polarization at the groovebecomes very weak. This is probably because when the object has a brightcolor, diffuse reflection components will prevail and make the specularreflection in two stages almost insensible. According to thisembodiment, in order to observe intense polarization, a polarized lightsource, of which the wavelength falls within a range where the spectralreflectance of the object is low, is used. For example, if the object isyellow, then a dark image is shot using a blue-based (i.e., thecomplementary color of the color yellow) light source color. Thewavelength range of the illuminating light can be determined with thespectral reflectance of the object taken into account.

FIG. 29 shows an exemplary spectral reflectance of the mucosa of a largeintestine, which is a typical organ to be observed with an endoscope.The large intestine mucosa has an outstanding peak of absorption in thewavelength range of 520 to 600 nm. That is why if the characteristic ofthe spectral filter is matched to that low-reflectance range asindicated by the reference numeral 2801, then the object will be shot ina relatively dark color in that wavelength range. Consequently, thesurface groove can be observed effectively with polarization asdescribed above according to the present invention. Also, for thatpurpose, the filter 2801 is arranged at the output stage of the lightsource 104. Optionally, the filter 2801 may be used selectively onlywhen a normal color image is not going to be shot. Also, the colorfilter could be arranged right before the image sensor 110.

Embodiment 3

Hereinafter, a third embodiment of an image processing apparatusaccording to the present invention will be described with reference toFIGS. 30A and 30B. The image processing apparatus of this embodiment isapplicable to not only an endoscope but also a camera with illuminationfor medical purposes (which may be used in a dermatologist's or adentist's), a fingerprint scanner, an optical surface analyzer and otherdevices.

FIG. 30A illustrates an exemplary configuration according to thisembodiment. The image processing apparatus of this embodiment uses adevice 400 instead of the endoscope 101 shown in FIG. 1C. The device 400includes a ring light 4001, a ring plane of polarization control element4002, a shooting lens 4003 and an image sensor 4004.

FIG. 30B illustrates a general appearance of the device shown in FIG.30A. In this embodiment, the ring plane of polarization control element4002 is arranged on the ring light 4001. A non-polarized light ray isinput through a light guide such as an optical filter to the ring light401 and the plane of polarization control element 4002 and has its planeof polarization rotated in the order of 0, 45, 90 and 135 degrees asshown in FIG. 2.

Alternatively, the ring light 4001 may also be a self-emitting lightsource such as an LED without using such a light guide that propagatesthe light emitted from a light source. Also, if the angle defined by theoptical axis of the image sensor with respect to that of theilluminating light is 15 degrees or less, the ring light may also bereplaced with a strobe light. However, if the ring light is used, evenan object that would be hard to observe with only one light can alsohave its surface microfacets and grooves estimated with high precision.Among other things, in that case, since the optical axis of theilluminating light can be substantially aligned with that of the imagesensor and can also be uniform, this device can be used effectively as adevice for scanning the surface of a product for any scratches orchecking its microfacets, a fingerprint scanner, or a skin unevennesschecker for a dermatologist. As for the image sensor 4004 and the imageprocessing processor (not shown), the image processing processor of thefirst embodiment can be used.

In the embodiment described above, the plane polarized light as thelight source is supposed to be rotated on a 45-degree-a-time basis.However, the angle of rotation does not have to be the same, but may bechanged, each time. On top of that, the angular interval does not haveto be 45 degrees, either. Nevertheless, in order to determine threeparameters for a cosine function, at least three samples are needed.That is to say, as for plane polarized light as a light source, itsangle of rotation can be changed into three or more different values.For example, if three sample angles are used, the three angles of 0, 60and 120 degrees may be selected, for example.

The present invention is broadly applicable to the field of imageprocessing that needs observing, checking, or recognizing the object'ssurface microfacets using a medical endoscope camera, a medical camerafor dermatologists, dentists, internists or surgeons, an industrialendoscope camera, a fingerprint scanner, or an optical surface analyzer.

While the present invention has been described with respect to preferredembodiments thereof, it will be apparent to those skilled in the artthat the disclosed invention may be modified in numerous ways and mayassume many embodiments other than those specifically described above.Accordingly, it is intended by the appended claims to cover allmodifications of the invention that fall within the true spirit andscope of the invention.

1. An image processing apparatus comprising: a polarized light sourcesection that sequentially illuminates an object with three or more kindsof plane polarized light rays, of which the planes of polarization havemutually different angles; an image capturing section that sequentiallycaptures an image of the object that is being illuminated with each ofthe three or more kinds of plane polarized light rays and directlyreceives light that is returning from the object by way of nopolarizers, thereby getting an intensity value; a varying intensityprocessing section that obtains a relation between the angle of theplane of polarization and the intensity value of each pixel based on asignal representing the intensity value supplied from the imagecapturing section, thereby generating an intensity maximizing angleimage that is defined by the angle of the plane of polarization thatmaximizes the intensity value with respect to each said pixel and adegree of intensity modulation image that is defined by the ratio of theamplitude of variation in the intensity value caused by the change ofthe plane of polarization to an average intensity value with respect toeach said pixel; and a normal estimating section that estimates, basedon the intensity maximizing angle image and the degree of intensitymodulation image, a normal to a tilted surface in a V-groove on theobject's surface on a pixel-by-pixel basis.
 2. The image processingapparatus of claim 1, wherein the normal estimating section includes: anazimuth angle processing section that obtains candidates for the azimuthangle of the normal based on the intensity maximizing angle image; azenith angle processing section that obtains the zenith angle of thenormal based on the degree of intensity modulation image; and an azimuthangle ambiguity processing section that chooses one among thosecandidates for the azimuth angle of the normal.
 3. The image processingapparatus of claim 1, comprising a normal image generating section thatgenerates an image of the normal that has been estimated by the normalestimating section.
 4. The image processing apparatus of claim 1,wherein the azimuth angle ambiguity processing section chooses one amongthose candidates for the azimuth angle of the normal by reference toeither a non-polarized light intensity image corresponding to an imageunder non-polarized light or the degree of intensity modulation image.5. The image processing apparatus of claim 4, wherein the varyingintensity processing section adds together the multiple light intensityimages that have been obtained by the image capturing section andcalculates their average, thereby generating and giving thenon-polarized light intensity image to the azimuth angle ambiguityprocessing section.
 6. The image processing apparatus of claim 4,wherein the azimuth angle ambiguity processing section chooses one amongthose candidates for the azimuth angle of the normal based on at leastone of the respective spatial gradient vectors of the non-polarizedlight intensity image and the degree of intensity modulation image. 7.The image processing apparatus of claim 1, wherein the polarized lightsource section and the image capturing section are attached to anendoscope.
 8. The image processing apparatus of claim 1, wherein thepolarized light source section gets non-polarized light transmittedthrough a plane of polarization changer that is able to change planes ofpolarization, thereby radiating plane polarized light rays, of which theplane of polarization sequentially changes into one of three or moredifferent types after another.
 9. The image processing apparatus ofclaim 1, wherein an angle of 15 degrees or less is defined between therespective optical axes of the polarized light source and the imagecapturing section.
 10. The image processing apparatus of claim 1,wherein the image capturing section includes either a monochrome imagesensor or a color image sensor.
 11. The image processing apparatus ofclaim 1, comprising: an illuminating direction setting section thatvirtually changes freely the illuminating direction of the object; and alight intensity image generating section that generates, based on theoutput of the normal estimating section, a light intensity image of theobject being illuminated in the illuminating direction.
 12. The imageprocessing apparatus of claim 1, wherein the polarized light sourcesection includes, on its output stage, a spectral filter that transmitslight falling within a wavelength range associated with a reflectance atwhich the spectral reflectance characteristic at the object's surfacereaches a local minimum.
 13. The image processing apparatus of claim 1,wherein the polarized light source section includes: a ring light sourcethat radiates non-polarized light; and a ring plane of polarizationchanger that transforms the non-polarized light radiated from the ringlight source into the plane polarized light ray and that is able tochange the angle of the plane of polarization of the plane polarizedlight ray sequentially.
 14. An image processing method comprising thesteps of: sequentially illuminating an object with three or more kindsof plane polarized light rays, of which the planes of polarization havemutually different angles; sequentially capturing an image of the objectwhen the object is being illuminated with each of the three or morekinds of plane polarized light rays and directly receiving light that isreturning from the object by way of no polarizers, thereby getting anintensity value; obtaining a relation between the angle of the plane ofpolarization and the intensity value of each pixel, thereby generatingan intensity maximizing angle image that is defined by the angle of theplane of polarization that maximizes the intensity value with respect toeach said pixel and a degree of intensity modulation image that isdefined by the ratio of the amplitude of variation in the intensityvalue caused by the change of the plane of polarization to an averageintensity value with respect to each said pixel; and estimating, basedon the intensity maximizing angle image and the degree of intensitymodulation image, a normal to a tilted surface in a V-groove on theobject's surface on a pixel-by-pixel basis.
 15. An image processingprocessor that receives a plurality of polarized images, where the planeof polarization of a plane polarized light ray with which an object isilluminated has three or more different angles, and that estimates,through image processing, a normal to a tilted surface in a V-groove onthe object's surface on a pixel-by-pixel basis, wherein the imageprocessing processor performs the steps of: obtaining a relation betweenthe angle of the plane of polarization and the intensity value of eachpixel based on the polarized images, thereby generating an intensitymaximizing angle image that is defined by the angle of the plane ofpolarization that maximizes the intensity value with respect to eachsaid pixel and a degree of intensity modulation image that is defined bythe ratio of the amplitude of variation in the intensity value caused bythe change of the plane of polarization to an average intensity valuewith respect to each said pixel; and estimating, based on the intensitymaximizing angle image and the degree of intensity modulation image, anormal to the tilted surface in the V-groove on the object's surface ona pixel-by-pixel basis.